In computing, a data stream refers to a sequence of data elements that a system may receive over time. Often the data elements are processed one at a time as the data stream is received. This is quite unlike batch data processing, in which a finite amount of data is received in batches and the finite amount of data is processed as a whole per batch. This is also quite unlike bulk data processing, in which a finite amount of data is received at once and then processed as a whole. That is, normal computer functions cannot operate on data streams as a whole. Rather, processing data streams requires a steady high-speed rate sufficient to support applications that consume the data streams and a queuing or buffering mechanism to make sure that enough data is being continuously received without any noticeable time lag.
To this end, U.S. Pat. No. 7,127,520 utilizes queuing in a data transformation solution for transforming an input data stream in a first data format to an output data stream in a second format. The input data stream is stored in one of a plurality of input queues. The input queues are operatively connected to job threads. The job threads, independent of one another and in parallel, receive input data streams from the input queues, format the input data steams into output data streams, and store the output data streams in the output queues, from where output connectors can pick them up and pass them on to their final destination. The use of queuing is one embodiment of the system disclosed in U.S. Pat. No. 7,127,520.
U.S. Pat. No. 8,914,809 provides another solution for processing data streams. In this case, a message broker can be configured to receive the message instances output as input message instances, store the input message instances in a message store, alter the structure or content of the input message instances according to a message brokering model to generate output message instances and provide output message instances to one or more downstream processors. This solution allows for autonomous storage and alteration of messages generated when processing a data stream of a first format to transform it into a second format.
With the amount of data produced by data streams growing continuously in today's world, management and processing of data stream have become exceedingly complex. Accordingly, there is a continuing need for innovations and improvements.